Perf: Optimize pages loading (Filecache path like approach) by Koc · Pull Request #2549 · nextcloud/collectives

Koc · 2026-05-31T13:14:08Z

📝 Summary

This is alternative approach to #2390 that fixes same performance issue (closes #2380).

Benefits comparing to previous implementation:

no extra columns
no migration to re-process already existent page
no listeners
much simpler implementation

So, we're just load all necessary pages via simple query SELECT * FROM filecache WHERE storage_id = <storageId> AND path LIKE 'appdata_<instanceId>/collectives/<collectiveId>/%'

🖼️ Screenshots

Collective with 390 pages with various nesting level

🏚️ Before	🏡 After
747 queries - depends on pages count and nesting level	50 queries - more or less constant

🚧 TODO

As a future improvement we can consider to add index to filecache table to storage_id, path columns (but this requires extra PR to nextcloud/server)

🏁 Checklist

Code is properly formatted (npm run lint / npm run stylelint / composer run cs:check)
Sign-off message is added to all commits
Tests (unit, integration and/or end-to-end) passing and the changes are covered with tests
Documentation (README or documentation) has been updated or is not required

🤖 AI (if applicable)

The content of this PR was partly or fully generated using AI tools
The AI-generated content was reviewed, comprehended and tested by a human

Signed-off-by: Kostiantyn Miakshyn <molodchick@gmail.com>

mejo-

Thanks a lot @Koc, this looks really promising 🤩

I have some comments, but I'm genuinely curious what you think about the comments.

If you do further changes to the PR, could you do them in separate fixup commits (and don't force-push changes to the existing commit for now) so it's easier to review your changes?

mejo- · 2026-06-09T10:13:43Z

+	 * @throws NotFoundException
+	 * @throws NotPermittedException
+	 */
+	public function getPagesFromFolderV2(int $collectiveId, Folder $folder, string $userId, bool $recurse = false, bool $forceIndex = false): array {


Is there a reason to introduce this as a new function instead of replacing getPagesFromFolder() altogether? There's still one call left to the old function (lib/Service/PageService.php:984), was this left out on purpose?

I'd prefer to replace the old function and update the Unit tests along, as it would be great to not loose the existing unit testing for getPagesFromFolder() for the new function.

I would say that this is temporary POC method to have possibility quickly switch between old an new implementation. So it's easier to add V2 prefix to call.

But, anyway, I guess that old implementation should be preserved for background jobs/console commands to re-create tree.

mejo- · 2026-06-09T10:16:26Z

+ * `subfolder/page.md`), so it matches the semantics of `File::getInternalPath()`
+ * within the jailed collective storage.
+ */
+class FileInfo {


Maybe better to call this CollectiveFileInfo to not confuse it with the public FileInfo API from server.

mejo- · 2026-06-09T10:26:04Z

+		$qb->select('fileid', 'storage', 'path', 'parent', 'name', 'mimetype', 'mimepart',
+			'size', 'mtime', 'storage_mtime', 'encrypted', 'etag', 'permissions')
+			->from('filecache')
+			->where($qb->expr()->eq('storage', $qb->createNamedParameter($storageId, IQueryBuilder::PARAM_INT)))


Any reason to not further filter on mimetype here and only get folders and Markdown files? If I understand the code further down correctly, we only process folders and Markdown files anyway, right?

My goal was to introduce a new index that will cover columns used in filter. So, in this case we should extend it with mime type then

mejo- · 2026-06-09T10:28:13Z

+				$pageFileIds[] = $fileInfo->fileId;
+			}
+		}
+		$pagesByFileId = $this->pageMapper->findByFileIds($pageFileIds);


I wonder whether this could become expensive if a few hundred fileIds get passed here. Not sure how well DB backends work with a large list of parameters to IN(). Did you test it on a collective with 500+ pages?

Well, here the limit of max 999 (SQLite) or 1000 (Oracle) hits. So we'd have to chunk these queries. Probably best done inside PageMapper::findByFileIds and PageLinkMapper::findByPageIds so callers don't need to know. But can be done in a follow-up later on.

Yeah, it should be fine from performance perspective. I will add workaround for 1000+ items (this is a known issue from other modules 😄 )

mejo- · 2026-06-09T10:40:27Z

+			}
+
+			// Create missing index page if folder or subfolders have page files (or forceIndex)
+			$subFolder = $collectiveFolder->getFirstNodeById($folderId);


I found the variable name subFolder missleading, as it's the folder object of the current folder. Why not call it folder instead?

mejo- · 2026-06-09T10:43:06Z

+	 * @throws NotFoundException
+	 * @throws \OCP\DB\Exception
+	 */
+	public function getFileCacheForCollective(int $collectiveId): array {


Wouldn't it be an option to pass the folder name here as well and further narrow down the query to only contain files in this subfolder? This would make the query less heavy when getPagesFromFolderV2() is called with a subfolder and not for the whole collective folder.

This would be particularly helpful for building the templateFolder page tree, as that one doesn't need to process all the files from the collective rood folder.

mejo- · 2026-06-09T10:49:53Z

+		$this->setTimestamp($fileInfo->mtime);
+		$this->setSize($fileInfo->size);
+		$this->setFileName($fileInfo->name);
+		if ($collectivePath !== null) {


I guess these checks can be omitted as the default is null anyway, so setting them to null again doesn't hurt.

mejo- · 2026-06-09T11:04:57Z

+	/**
+	 * Build the page info from a lightweight filecache entry (see PageService::getPagesFromFolderV2).
+	 */
+	public function fromFileInfo(


This copies much of the logic of fromFile(). Maybe worth refactoring to have less code duplication?

Koc requested review from max-nextcloud, mejo- and silverkszlo as code owners May 31, 2026 13:14

Koc mentioned this pull request May 31, 2026

Optimize performance of the pages loading by stopping using filesystem always #2380

Open

Koc force-pushed the feature/optimize-pages-loading-v2 branch 3 times, most recently from e644e8e to 5d10baa Compare May 31, 2026 13:26

Perf: Optimize pages loading

5e6b2e9

Signed-off-by: Kostiantyn Miakshyn <molodchick@gmail.com>

Koc force-pushed the feature/optimize-pages-loading-v2 branch from 5d10baa to 5e6b2e9 Compare May 31, 2026 13:31

mejo- requested changes Jun 9, 2026

View reviewed changes

Koc self-assigned this Jun 9, 2026

Conversation

Koc commented May 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📝 Summary

🖼️ Screenshots

🚧 TODO

🏁 Checklist

🤖 AI (if applicable)

Uh oh!

mejo- left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Koc commented May 31, 2026 •

edited

Loading